Image processing method for stimulating structure in relief, and device for carrying out the method

ABSTRACT

The invention concerns an image-processing method and system for the simulation of structure in relief in image of objects having at least one polygonal surface. The polygonal surface is divided up into a large number of raster elements, and for each raster element, an inclination value determining the inclination of the local surface normal is read out of relief-structure matrix. From the inclination value and either the normal to the polygonal surface or the local normal to the surface, the inclined local normal is calculated and the brightness and/or color of each raster element calculated as a function of the direction of view of the observer, the illumination conditions and the angle of the inclined local normal.

DESCRIPTION

The invention relates to an image-processing method for simulating astructure in relief in representing objects that particularly compriseat least one polygon surface, as defined in the preamble to claim 1, andan image-processing apparatus, as defined in the preamble to claim 11,for executing the image-processing method.

In computer-graphics systems, bodies are usually simulated by gridmodels. Each body is described by the spatial coordinates of the nodepoints of the grid model. For display on a screen, the spatialcoordinates from the three-dimensional coordinate system are converted,for example through central projection, into a two-dimensionalscreen-coordinate system.

The surface elements between the node points are usually represented asideally smooth surfaces without uneven areas. Natural surfaces are,however, not smooth, but are relatively uneven and structured. Forexample, the surface of human skin has a fine porous structure. Thesurfaces represented with conventional computer-graphics systems aretherefore perceived as unnatural by the viewer. To create a naturalsurface impression, the grid model would have to be refined soextensively that the spacing between the node points would be smallerthan the uneven areas to be modeled. This would place significantly moredemanding requirements on the storage capacity and computing capabilityof the computer-graphics system.

A method of simulating a relief structure in representing such surfacesis known from CATMULL, E. E.: Computer Display of Curved Surfaces, Proc.IEEE Conf. on Computer Graphics, Pattern Recognition and DataStructures, Los Angeles (May 1975), 11. In this case, a two-dimensionalimage of the desired relief structure--hereinafter referred to astexture image--is superposed over each surface element of the gridmodel. The brightness of a surface element is not consistent, but varieswithin the surface element corresponding to the texture image.

If the viewer looks at the surface element from the same direction fromwhich the texture image was also received, and if the illumination isalso the same as in the receiving of the texture image, the viewerobtains a natural visual impression of the relief structure of thesurface.

In this case, the viewer always perceives the same texture image,regardless of the perspective and the illumination direction. Thedependencies on the angle and direction of the reflection behavior ofthe relief structure of a natural surface are therefore not simulated inthis method. The surface thus appears as a smooth surface onto whichonly a type of photo is placed. Such representations have an unnaturaleffect similar to that of objects to which a decorative sheet is affixedto give the appearance of a surface structure.

A method of simulating a relief structure in representing surfaces,which also takes into consideration the perspective of the viewer andthe illumination direction, is further known from BLINN, J. F.:Simulation of Wrinkled Surfaces, Computer Graphics, 12(3), pp. 286-292,Proc. SIGGRAPH '78.

In this method, the surface normal that is decisive for the reflectionbehavior of the surface element is first calculated for each surfaceelement. Furthermore, a relief function is defined, which defines theelevations and depressions--in other words, the relief structure of thesurface--at each point of the surface element. The local normal vectorfor each point of the surface element is then calculated from thesurface normal and the relief function through partial derivations.Because of the dependency of the reflection behavior on angle anddirection, the brightness varies within the surface element as afunction of the position of the local surface normals. Consequently, therelief structure of the surface can be simulated with consideration ofthe locally-alternating illumination and viewing directions and theresulting, different reflection behavior of the surface.

This method creates a relatively natural impression of the surface forall positions of the object. However, a disadvantage of the method isthat partial derivations of the relief functions must be formed fordetermining the local brightness within a surface element, whichrequires a substantial amount of calculation effort. This lengthens thetime span necessary for calculating an image, which hinders a real-timerepresentation of movement, particularly with varying light conditions.

It is therefore particularly the object of the invention to provide animage-processing method for simulating a relief structure in therepresentation of computer-modeled surfaces of an object, as well as anapparatus for executing the image-processing method, which also permitsrapid, real-time representations of movement.

The object is accomplished by the features disclosed in claim 1,and--with regard to the image-processing apparatus--by the features ofclaim 11.

The invention includes the technical teaching of dividing each polygonsurface--as a reference surface--of an object to be represented into aplurality of pixels, the polygon surface being defined by the nodepoints of the object represented as a grid model; determining aninclined local surface normal--hereinafter referred to as pixelnormal--for each pixel from an original relief value that defines therelief structure of the surface and either the surface normal of thepolygon surface or a local surface normal; and calculating thebrightness of the pixel with the use of a local illumination model as afunction of the position of the pixel normal relative to theillumination direction and the viewing direction.

Thus, the precise spatial position of each surface element need nolonger be re-ascertained in all subsequent spatial movements performedby the object. The treatment of the simplified "wire model" issufficient here. The spatial structure of the surface is updated basedon the reference surfaces, without consideration of the actual spatialposition of the pixels in relation to the reference surface. This isfurther treated to some extent as a two-dimensional structure, but thisstructure does not remain unchanged, as in a photographic image rotatedin space; instead, the information that changes when the viewingdirection changes is purposefully changed. The surface normal associatedwith each pixel and representing the offset of the orientation of thepixel surface with respect to the orientation of the reference surfaceserves this purpose. With this value, it is possible, with littleeffort, to "match" the optical impression of the polygon surfaceoriented spatially differently.

The invention makes use of the knowledge that the optical impression ofthe relief structure of a surface is not determined so much from theactual (fine) position of the pixel in space, but from interaction withlight sources in the form of light and shadow formation, or reflectionat other objects. These actions are essentially dependent on the spatialposition of the local surface normal of the pixel relative to theviewing direction. This permits a very rapid, uncomplicated updating asa function of the viewer's perspective of the relevant illuminationdirection. In contrast, the local depressions and elevations of thesurface contour have only little impact on the optical impression of therelief structure, and can therefore be disregarded.

The invention includes all options that involve replacing an object tobe represented for display with an object having a simplified surfacestructure, with the fine surface structure being replaced by a piece oflocal information regarding the orientation of the surface within theframework of the fine structure, and the original fine structure beingreplaced by a plane image that is changed with consideration of theeffects of the illumination and viewing directions. In particular, theeffect of illuminating an object from the inside (luminous object) orthe influence of other effects (reflections) originating from otherobjects is also taken into consideration.

The method of the invention permits the simulation of a relief structurein the representation of a body formed by polygon surfaces. The spatialposition of the polygon surface is usually defined by the spatialcoordinates of the node points of a grid model that simulates the bodyor surface to be represented in the computer.

The reference surface need not be level in every case; as will be shownbelow, it can have one or a plurality of deformities (for examplebulges) that approximates the actual surface shape of the object to berepresented. An important point is that the surface shape of thereference surfaces is different from the fine structure of the object tobe represented, which difference is compensated by the measures of theinvention.

Wherever the invention is described below in conjunction with polygonsurfaces, this may refer to all of the elements of an object thatrepresent a surface segment, the elements having a reduced structureinformation in comparison to the fine structure of the surface to berepresented. For example, the surface segments can be convex, such asthe segments of a ball, to make the impacts, that is, the changes indirection, as small as possible at the boundaries of the segments in theregion of the surface. The invention encompasses all measures in which,in the representation, a spatial base structure of an object issuperposed with a fine surface structure containing information thatcharacterizes the local orientation of the surface and is used togenerate the optical effects of the orientation of illumination andviewing on the relevant surface element, or its vicinity.

A particular advantage of the invention is that the operations relevantto the structure, which are to be performed with the measures of theinvention, can be divided into repeating regions (modules), so therepeating portions of a structure permit an additional reduction in thecalculation effort.

The measures of the invention can be implemented with both purelysoftware-based measures and suitable special hardware. The advantage ofthe hardware-based solution is that considerable time advantages can beattained particularly through operations that are performedsimultaneously in assemblies disposed in parallel.

Wherever "assemblies" are discussed below, they may also be functionalgroups, for example in the form of electrical circuits. It is alsopossible that one and the same assembly or functional groupconsecutively performs different operations, as is generally the case inprocessor technology. As explained, however, speed advantages can beattained especially when functional groups are active adjacently and inparallel.

Within the scope of the measures of the invention for representingobjects within the framework of image processing, the individual polygonsurfaces are preferably divided into a plurality of parallel,equidistant scanlines in which the pixels are in turn disposedequidistantly.

The polygon surface is divided by the first assembly or functional groupof a calculation unit, which receives at its input a parameter setrepresenting the spatial position of the polygon surface, and whichoutputs a plurality of coordinate sets at its output, the setsrepresenting the spatial position of a respective pixel.

A relief value that defines the relief structure is then determined foreach pixel as a function of the position of the pixel within the polygonsurface.

In this connection, it is important that the relief value is solelydependent on the position of the pixel within the polygon surface, notthe global position of the pixel or the spatial position of the polygonsurface. This is necessary, because otherwise the relief structureappearing on the polygon surface would "migrate" on the surface during amovement of the polygon.

Therefore, a storage matrix--also referred to hereinafter as reliefmatrix--is provided, which contains the inclination values of the localsurface normal that result from the course of the surface for apredetermined relief structure. Each element of the storage matrix istherefore associated with a point of the relief structure, and containsan inclination value for this point of the relief structure. Forexample, the inclination vector can be stored as an inclination value,that is, the vector that must be added to the local surface normal ofthe smooth surface in order to create the impression of the reliefstructure. It is also possible to store a rotation matrix as aninclination value, with the matrix defining the rotation of the localsurface normal. This is particularly advantageous, because the length ofthe local surface normal is not changed by a rotation.

In this connection, it is important that the relief structure itself isnot stored as a relief in the relief matrix, but the inclination of thelocal surface normal resulting from the course of the surface is stored.The local surface normal that is inclined because of the reliefstructure can therefore be calculated advantageously quickly and withlittle calculation effort.

For each pixel, an inclination value that represents the inclination ofthe local surface of this pixel that has been caused by the reliefstructure is read out. The storage matrix is therefore addressed by thecoordinate set of the respective pixel.

In a preferred embodiment of the invention, therefore, for each positionof a polygon, a transformation matrix M is calculated that transformsthe spatial coordinates (x_(i), y_(i), z_(i)) of the pixels into atwo-dimensional coordinate system (U,V): ##EQU1##

The U, V value pair serves in addressing the relief matrix from whichthe relief value of the pixel is then read out. In this connection, itis important that the transformation matrix M be calculated individuallyfor each polygon, or each position of a polygon, so that the sameelement of the relief matrix, and therefore the same relief value, isalways associated with each pixel independently of the spatial positionof the polygon.

The transformation matrix M is calculated, for example, from the spatialcoordinates of the corner points of the polygon surface. If the polygonsurface is a triangle, for example, then the spatial coordinates (x_(i),y_(i), z_(i)) of the three corner points P₁, P₂, P₃ of the triangle areknown. As explained above, the mapping of the three-dimensionalcoordinate system into the U, V coordinate system must be independent ofthe position of the polygon surface so that the relief structuresimulated on the polygon surface does not migrate within the polygonsurface when the surface moves. This can be achieved, for example, bymapping the corner points P_(i) of a polygon surface onto certain pointsP'_(i) in the U, V coordinate system, independently of the position ofthe polygon surface. The points P'_(i) are first predeterminedarbitrarily. If a triangle is observed as the polygon surface, thefollowing applies for the mapping of the three corner points P_(i)(x_(i) y_(i) z_(i)) of the triangle of the three-dimensional X, Y, Zcoordinate system into the U, V coordinate system. ##EQU2##

Because the points P'_(i) (u_(i), v_(i)) are predetermined arbitrarily,an equation system having six linearly independent equations is known.From this system, the six elements M_(ij) of the transformation matrixcan be calculated in a simple manner, for example with the use ofCramer's Rule. The transformation matrix M is therefore usuallycalculated once for each polygon surface.

If the polygon surface is viewed in perspective, distortions tend tooccur if the viewer looks at the polygon surface at an acute angle,because in this case the corner points of the polygon surface are at adifferent distance from the viewer. In a preferred embodiment of themethod of the invention, therefore, homogenized coordinates x, y, z areused for defining the position of the corner points of the polygonsurface. For this purpose, the coordinates x*, y*, z* of the cornerpoints are respectively multiplied by the so-called homogeneouscomponent 1/w, which, for each corner point, is proportional to thedistance of this corner point from the viewer.

As explained above, the optical impression of the relief structure isessentially dependent on the position of the local surface normalrelative to the viewing direction and the illumination direction. Inthis variation of the invention, the relief value stored in the reliefmatrix therefore contains two components Δu, Δv, which define the localinclination of the local surface normal that is caused by the reliefstructure. Thus, the component Δu determines to what degree the localsurface normal is inclined in the direction of the U axis, while thecomponent Δv determines the inclination of the local surface normal withrespect to the V axis.

Not only is the inclination of the local surface normal caused by therelief structure decisive in the U, V coordinate system; thecorresponding inclination of the pixel normal is decisive in thespatial-coordinate system as well.

The aforementioned calculation unit therefore includes a secondassembly, which uses the first parameter set representing the spatialposition of the polygon surface, and/or the coordinate sets of theindividual pixels, to calculate a second parameter set, which representsthe spatial position of the surface normal of the polygon surface or thelocal surface normal of the individual pixels, without considering therelief structure.

The calculation unit further includes a third assembly, which uses theinclination value--in this variation, the components Δu, Δv--and thesurface normal of the polygon surface, or the local surface normal ofthe smooth surface, to calculate the inclined local surface normal. Forthis purpose, the input of the third assembly receives the coordinatesets of the pixels and the associated inclination values from the reliefmatrix, and uses them to calculate a third parameter set, whichrepresents the spatial position of the surface normal of the polygonsurface, taking into consideration the relief structure.

In a preferred variation of the invention, therefore, a furthertransformation matrix A is calculated for each polygon; this matrixpermits the calculation of the inclination of the pixel normal in thespatial-coordinate system from the components Δu, Δv.

For the pixel normal (n'_(x), n'_(y), n'_(z))^(T), ##EQU3## follows fromthe surface normal (n_(x), n_(y), n_(z))^(T) of the smooth surface andthe relief-structure values Δu, Δv.

According to the invention, different options are provided forcalculating the transformation matrix A, some of which are describedbriefly below.

In one variation of the invention, the transformation matrix A iscalculated from the transformation matrix M, a geometry-transformationmatrix G that is predetermined for the polygon surface, and a furthermatrix T.

First, the normalized normal vector n_(N) =(n_(xN) n_(yN) n_(zN))^(T) iscalculated from the surface normal of the polygon surface n=(n_(x) n_(y)n_(z))^(T) : ##EQU4## For the transformation matrix T, it follows that:##EQU5## or, written in terms of components: ##EQU6##

In the same way, the transformation matrix T_(L) is calculated from thelocal surface normal n_(L).

With the geometry-transformation matrix G, it then follows that:

    G'=G·T.sup.-1

From this, the transformation matrix A can be calculated directly:

    A=G'·T.sub.L ·M

or, written in terms of components: ##EQU7##

Relatively complicated and calculation-intensive vectorial operationsmust be performed in the above-described calculation of thetransformation matrix T or T_(L).

In another variation of the invention, therefore, a plurality oftransformation matrices T or T_(L) is calculated in advance fordifferent spatial positions of the surface normal, and each is stored inan element of a three-dimensional storage matrix S. Each element of thestorage matrix S therefore contains a complete transformation matrix Tor T_(L).

The storage matrix S simulates a cubic envelope surface, the centerpoint of which is the origin of the surface normal. A transformationmatrix T or T_(L) is associated with each point on the envelope surface.Therefore, for determining the transformation matrix T or T_(L), thepoint of intersection of the surface normal or its extension with thecubic envelope surface must be determined.

The first dimension of the storage matrix S is addressed as a functionof the side surface of the cubic envelope surface that is intersected bythe surface normal. The following applies for the addressing value a₁ ofthe first dimension of the storage matrix S: ##EQU8##

The other two dimensions of the storage matrix S are addressed as afunction of the position of the point of intersection within sidesurface penetrated by the surface normal. The following applies for theaddressing values a₂, a₃ : ##EQU9##

The transformation matrix is then read out of the storage matrix S:

    T=S(a.sub.1, a.sub.2, a.sub.3)

    or

    T.sub.L =S(a.sub.1, a.sub.2, a.sub.3)

The transformation matrix A is calculated in the same way as in theabove-described variation of the invention:

    A=G'·T.sub.L ·M

This variation advantageously permits a relatively rapid calculation ofthe transformation matrix A. The elements of the storage matrix S arepreferably calculated in the same way as the transformation matrices Tin the above-described variation.

In the above-described variation of the invention, the pixel normal thatis inclined because of the relief structure is calculated throughmultiplication of the inclination vector in the U, V coordinate systemby the transformation matrix A.

In another variation of the invention, the inclined pixel normal iscalculated with the aid of so-called quaternions. Quaternions arevectorial operations that include a scalar component and three vectorialcomponents. From the inclination vector

    d=[Δu, Δv, 00].sup.T

in the U, V coordinate system and the quaternion

    q.sub.i =(s.sub.i,(c.sub.x.sup.i, c.sub.y.sup.i, c.sub.z.sup.i)),

for the inclined pixel normal (n'_(x), n'_(y), n'_(z))^(T) in the X, Y,Z coordinate system, the following results from the pixel normal (n_(x),n_(y), n_(z))^(T) of the smooth polygon surface: ##EQU10##

The quaternions are calculated, for example, through interpolation ofthe quaternions that are predetermined for each pixel in the cornerpoints of the polygon surface. The interpolation can advantageously beperformed component-wise, and therefore only requires a small amount ofeffort. In this case, a quaternion is predetermined for each cornerpoint of the polygon surface, and the quaternions of the individualpixels are then calculated from this quaternion.

The relief structure effects an inclination of the pixel normal withrespect to the surface normal of the smooth surface. The inclination ofthe pixel normal caused by the relief structure can be oriented eitherwith respect to the surface normal of the entire polygon, or withrespect to the pixel normal.

If the inclination of the pixel normal because of the relief structureis oriented with respect to the surface normal of the polygon surface, arelief structure can only be simulated correctly on planar polygonsurfaces, because in such polygon surfaces, the surface normal is onlyconsistent within the polygon surface. The calculation of the surfacenormal of the polygon surface can be determined from the spatialcoordinates of the corner points of the polygon surface. For thispurpose, for example, the edge vectors of the polygon surface arecalculated through component-wise subtraction of the spatial coordinatesof adjacent corner points. The edge vectors determined in this waynaturally lie in the plane defined by the polygon surface. The surfacenormal then results as a vector product of two edge vectors.

In curved polygon surfaces, in contrast, the surface normal varieswithin the polygon surface. The inclination of the pixel normal causedby the relief structure must therefore be oriented with respect to thelocal surface normal of the smooth, but curved, polygon surface.Therefore, the local surface normal of the smooth polygon surface--thatis, the surface having no relief structure--must first be calculated forall pixels. This is preferably effected through interpolation of thecorner-point normals. To this end, in curved polygon surfaces, the localsurface normals are predetermined in the corner points of the polygonsurface. The local surface normal at an arbitrary point on the polygonsurface results from the interpolation of the corner-point normals.

The predetermining of the corner-point normals further ensures a"gentle" transition between adjacent polygon surfaces, because acorner-point normal is respectively part of a plurality of adjacentpolygon surfaces, and the surface is therefore smooth at the connectingline between two adjacent polygon surfaces.

In the interpolation of the local surface normal of the curved polygonsurface from the corner-point normals, a normal vector representing anintermediate position must be determined as a function of, on the onehand, the position of the pixel to be observed, the pixel being locatedbetween the corner points of the polygon surface, and, on the otherhand, as a function of the corner-point normals. If an interpolation isto be performed between only two corner-point normals, that is, on aconnecting line between the two corner points, the one corner-pointnormal can be mapped through a shift and a rotation toward the othercorner-point normal. A problem in the interpolation of the local surfacenormal is that a component-wise interpolation cannot be performed simplyin a mapping comprising a rotation and a shift.

In an advantageous variation of the invention, therefore, the localsurface normal is interpolated with the aid of the so-calledquaternions. Quaternions are four-dimensional vectorial operators havinga scalar component and three vectorial components. A quaternion ispredetermined for each corner-point normal. An arbitrary local surfacenormal can advantageously be interpolated through component-wiseinterpolation of the quaternions. If, for example, the local surfacenormal is to be calculated in the center between two corner points, andthe interpolation is based solely on these two corner points, thequaternion of this local surface normal, resulting in this case as theaverage value of the quaternions predetermined for the two corner-pointnormals, is to be determined. The local surface normal then results fromthe linking of the one corner-point normal with the calculatedquaternion.

Hence, the spatial position of the local surface normal (pixel normal)and the position of the pixel are known for each pixel of the polygonsurface. The method of the invention therefore provides the calculationof a brightness value or color value for each pixel, based on a localillumination model. As described in detail above, the brightness that aviewer perceives a pixel to have is essentially dependent on theposition of the pixel normal relative to the illumination direction andthe viewing direction, that is, the perspective of the viewer.

The aforementioned calculation unit includes a fourth assembly forcalculating the brightness value and color value. At its output, thefourth assembly receives the third parameter set, which represents thespatial position of the local, inclined surface normal. The brightnessof a pixel is therefore determined by the orientation of the localsurface normal of the polygon surface, taking into consideration therelief structure.

The fourth assembly preferably also receives a predetermined, fourthparameter set, which represents the spatial position of the viewer, and,possibly, a fifth parameter set representing the illuminationconditions.

In a variation of the invention, it is significant enough to meritprotection that diffuse reflection at the pixel is assumed forcalculating the brightness of the pixel. The light impacting the pixelis radiated uniformly in all directions. The brightness of a pixel thatis perceived by a viewer is therefore basically a function of theorientation of the pixel normal with respect to the illuminationenvironment. Light that is incident parallel to the pixel normal ispreferably reflected. If the direction of light incidence differs fromthe pixel normal, the light is reflected correspondingly less intensely.Therefore, each pixel can be allocated a reflection characteristic thatrepresents the dependence of the pixel brightness on the direction andangle of the light incidence.

To define the reflection characteristic, in this variation of theinvention, the angular differences between the pixel normal and theillumination direction are calculated in two different, preferablyperpendicular, planes. The reflection behavior is then calculated fromthese two angular differences. In a preferred embodiment of thisvariation, the two angular differences serve in addressing atwo-dimensional matrix--referred to hereinafter as reflectionmatrix--that contains the angle- and direction-dependent reflectionfactors. In this way, it is possible to define an arbitrary reflectioncharacteristic of the pixel.

In another variation of the invention, the brightness of the pixel iscalculated as a function of the position of the pixel normal relative tothe viewing direction. This embodiment utilizes the knowledge that thepixel appears brightest when the pixel normal and the viewing directionextend parallel, because the maximum amount of the pixel surface of thepixel is then visible. If the pixel normal is inclined with respect tothe viewing direction, the pixel appears to be correspondingly darker.Thus, it is possible to calculate a reflection characteristic thatrepresents the brightness of a pixel as a function of the position ofthe pixel normal relative to the viewing direction.

To define the reflection characteristic, the angular differences betweenthe pixel normal and the viewing direction are calculated in twodifferent, preferably perpendicular, planes. The reflection factor ofthe pixel is then calculated from these two angular differences.

In a preferred embodiment of this variation, the two angular differencesserve in addressing a two-dimensional matrix in which the reflectionfactor for different values of the angular differences is stored.

In another variation of the invention, it is significant enough to meritprotection that mirror reflection at the pixel is assumed forcalculating the brightness of the pixel. The light impacting the pixelis reflected at the pixel according to the laws of geometrical optics,with the angle of incidence and the angle of reflection being identical.

Conversely, if a "line of sight" originating from the viewer isobserved, rather than a beam of light originating from a light source,this "line of sight" is reflected at the pixel in the direction of areflection vector. The spatial position of this reflection vectorlikewise results from the fact that the angles of incidence andreflection are identical according to the laws of geometrical optics.

An optimum viewing condition for the viewer is for the reflection vectorto extend parallel to the direction of light incidence, because then theviewer has an optimum view of the light source and the pixel appears tobe brightest. If, however, the reflection vector differs from thedirection of light incidence, the reflection condition is not metexactly and the pixel appears to be correspondingly darker.

A reflection characteristic can be defined that represents thedependence of the reflection on angle and direction. In contrast todiffuse reflection, the reflection factor here is also dependent on theviewing direction, and not solely on the position of the pixel normalrelative to the illumination direction.

To calculate the reflection characteristic, the angular differencesbetween the reflection vector and the illumination direction arecalculated in two different, preferably perpendicular, planes. Thereflection factor for the mirror reflection is then calculated fromthese angles. In a preferred embodiment of this variation, the twoangles serve in addressing a reflection matrix that defines thereflection characteristic of the pixel.

The two aforementioned variations can also be combined through thecalculation of a unified reflection factor from the reflection factorfor diffuse reflection and the reflection factor for mirror reflection.The two reflection factors can be weighted differently. In this way, itis possible to simulate a natural reflection behavior.

In another variation of the invention, it is significant enough to meritprotection that a cubic envelope surface is placed around the pixel forcalculating the brightness of the pixel, with a value for the brightnessof the pixel being associated with each surface element of the envelopesurface. The envelope surface therefore contains an all-around recordingof the environment from the perspective of the pixel.

As in the above-described variations of the brightness calculation,diffuse reflection or mirror reflection can be selectively assumed.

If diffuse reflection is assumed, according to the method of theinvention, the point of intersection of the pixel normal or itsextension with the envelope surface is determined. The value of theenvelope surface that is associated with this point of intersection thenforms the brightness value of the pixel.

If, in contrast, mirror reflection is assumed at the pixel, the point ofintersection of the reflection vector with the envelope surface isdetermined, and the associated value of the envelope surface is used asa brightness value of the pixel.

In a preferred embodiment of this variation, a three-dimensionalmatrix--hereinafter referred to as reflection matrix--serves insimulating the cubic envelope surface and in defining the reflectionfactors for the different illumination and viewing directions. The cubicenvelope surface has six side surfaces, which are respectively assigneda distribution of values of the reflection factor. First, it isdetermined which side surface of the envelope surface is penetrated bythe pixel normal or reflection vector, with each side surface beingallocated a value that addresses the first dimension of the reflectionmatrix. The other two dimensions of the reflection matrix are addressedby the angles of the pixel normal or reflection vector with respect tothe axis of the coordinate system penetrating the side surface. In otherwords, the two other dimensions of the reflection matrix are addressedas a function of the position of the point of intersection on the sidesurface.

In an advantageous embodiment of the method of the invention, it isprovided that each polygon surface is superposed with a two-dimensionaltexture image in addition to the relief structure. Consequently, coloror brightness patterns of natural surfaces can be simulated. Superposingthe texture image can be effected in the same manner as superposing therelief structure. Hence, a value for the texture image is associatedwith each pixel of the polygon surface. An important point is that thevalue for the texture image is solely dependent on the position of thepixel within the polygon surface, not on the spatial position of thepolygon surface. Otherwise, the texture image would "migrate" within thepolygon surface if the surface moved.

The above-described method permits a simulation of a relief structure inthe representation of a polygon surface in that the local surface normalwithin the polygon surface is inclined with respect to the surfacenormal of the entire polygon surface as a function of a predeterminedrelief structure.

A prerequisite for this is that the polygon surface be completely leveland therefore have a consistent surface normal that can serve as areference value for the orientation of the local surface normal forsimulating the relief structure.

In modeling sharply-curved surfaces, it is often desirable to adapt theshape of the polygon surface to the shape of the surface to berepresented. The surface normal of the polygon surface is not consistentin this case, but varies within the polygon surface. This is notattributed to the inclination of the local surface normal because of therelief structure, but is only a function of the shape of the smoothsurface.

The surface normal is predetermined, for example, in the corner pointsof the polygon. The local surface normal within the polygon surface isthen calculated through interpolation of the corner-point normals.

The inclination of the surface normal because of the relief structure isoriented with respect to the local surface normal obtained throughinterpolation, and not--as with the assumption of a planar polygonsurface--with respect to the surface normal of the entire polygonsurface.

In the above-described variations of the image-processing method of theinvention, the relief structure simulated on the surface of the objectis constant over time. Often, however, it is desirable to simulate arelief structure that varies over time, for example to permit thesimulation of a change in the surface structure. In an advantageous,refining variation of the invention, therefore, the inclination valuethat is read out of the relief matrix is multiplied by a scaling factorprior to the calculation of the local, inclined surface normal to permita change to the spatial relief effect of the relief structure to besimulated. If the scaling factor equals zero, the surface of the objectto be represented appears to be completely smooth. As the scaling factorincreases, the relief effect of the simulated relief structure thatappears on the surface also increases. In this way, the relief effect ofthe relief structure can be set arbitrarily. It is also possible tosimulate the formation of a relief structure by increasing the scalingfactor gradually from zero during the representation.

In a contrasting variation of the invention, for simulating a dynamicrelief structure, a plurality of relief matrices is provided, whichmatrices respectively reflect an instantaneous recording of the reliefstructure to be simulated. During the representation of the spatialscene, the individual relief matrices are selected consecutively forcreating a dynamic image impression, and are used to calculate theinclined, local surface normal. Because the contents of the individualrelief matrices can be calculated in advance, hardly any additionalcalculating effort is necessary during the actual representation, whichis particularly critical in a real-time representation of courses ofmovement. It is only necessary that the desired relief matrix beselected and connected to the assembly that calculates the inclined,local surface normal from the surface normal of the smooth surface andthe inclination value.

In the same way, the texture appearing on the object's surface can besimulated dynamically. In an advantageous variation of theimage-processing method of the invention, therefore, numerous texturememories are provided, which respectively represent an instantaneousrecording of the texture image to be simulated on the object's surface.To create a movement impression, it is only necessary to switch betweenthe individual texture memories during the representation. Scarcely anyadditional calculation effort is needed during this representation,because the contents of the texture memory can be calculated in advance.

Because the texture memory usually contains the texture image directlyin digitized form, it is advantageously possible to provide theimage-replication memory of a video- or computer-graphics system as atexture memory, or to connect the texture memory to this type of systemin some other manner. In this way, it is possible to use conventionalvideo- or computer-graphics systems to simulate arbitrary animated imagesequences as a texture image on the object's surface.

As described in detail above, the image-processing method of theinvention provides a plurality of storage matrices (texture matrix,relief-structure matrix, reflection matrix for mirror reflection anddiffuse reflection, environment matrix) that influence the mappingbehavior of the image-processing method, and determine the appearanceimage of the spatial scene to be represented. The method described abovefor dynamically simulating relief structures or textures can be appliedin the same way to all storage matrices that influence the appearanceimage of the spatial scene. For example, interesting light effects canbe attained through dynamic changes in the reflection matrices.

The different options for dynamic simulation can be used alone or inrandom combination.

As described above, the body or surface to be displayed on the screen issimulated as a grid model in the computer. The surface thereforecomprises numerous polygon surfaces whose spatial position is defined bythe spatial coordinates of the node points of the grid model. In anadvantageous embodiment of the invention, triangles are used as thepolygon surfaces. This means that three points are respectivelyconnected by edges in the grid model. An advantage of using trianglesfor modeling the surfaces is that the surface normal of a triangle isdefined unambiguously by the spatial coordinates of the corner points,because three points always lie in a plane, while in polygons havingmore than three corner points, it is possible that the corner points donot lie exactly in a plane, so the surface normal is not definedexactly.

The above-described method is not limited to a monochromaticrepresentation. Rather, the method can be executed consecutively or inparallel for a plurality of colors, preferably the primary colors. Acolor-dependent reflection or absorption behavior of the surface can betaken into consideration.

In an advantageous variation of the invention, the apparatus isconfigured as an integrated circuit in a processor system, or as anautonomous processor, whose input receives a parameter set thatdetermines, among other things, the spatial position of a polygon. Atthe output, the spatial coordinates of individual pixels of the polygonsurface and the brightness value of the pixels are emitted, withconsideration of a predetermined relief structure. To represent thepolygon surface on a screen, the spatial coordinates of the individualpixels need merely be transformed in a conventional manner into atwo-dimensional screen-coordinate system. The circuit can thus beadvantageously incorporated into existing computer-graphics systems.

The solution according to the invention advantageously permits thesimulation of a relief structure in the representation of the polygonsurface of a grid model. The image-processing method of the inventionadvantageously permits a representation of the individual polygonsurfaces independently of one another. This means that it is alsopossible to represent a polygon surface without knowledge of the data ofother polygon surfaces. The image-processing apparatus can therefore beadvantageously constructed from a plurality of calculating unitsdisposed in parallel. This advantageously permits incorporation into aparallel calculation-unit arrangement, and therefore a high processingspeed.

Other advantageous modifications of the invention are characterized inthe dependent claims and described in detail below in the description ofthe preferred embodiment of the invention in conjunction with thefigures. Shown are in:

FIG. 1 as a preferred embodiment of the invention, a circuit, shown as ablock diagram, for simulating a relief structure,

FIG. 2 as a detailed block diagram, the fourth calculation unit fromFIG. 1,

FIG. 3a a triangle of the grid model with the associated parameters as aprojection onto the X-Y plane,

FIG. 3b the inclination of the triangle surface with respect to the Xand Y axes,

FIG. 4 a schematic diagram for calculating a relief matrix using as anexample a brick wall,

FIG. 5a a cubic envelope surface that surrounds the triangle forcalculating the local reflection factor in diffuse reflection at thepixel,

FIG. 5b a cubic envelope surface that surrounds the triangle forcalculating the local reflection factor in mirror reflection at thepixel,

FIG. 6 a circuit for simulating the cubic envelope shown in FIGS. 5a and5b,

FIG. 7 a circuit for calculating the inclination of the pixel normal asa function of the inclination in the U, V coordinate system with the aidof quaternions, and

FIG. 8 a diagram for explaining the scaling of a relief structure.

The circuit shown in FIG. 1 is used in imaging bodies or surfaces thatare present in the form of a grid model. In this instance, the surfaceof the body to be represented comprises a plurality of triangles. Thegrid model is therefore defined by the spatial coordinates of the cornerpoints.

The illustrated circuit serves particularly in simulating a reliefstructure of the surfaces.

For this purpose, the circuit receives, as an input value for eachtriangle, a parameter set that defines the spatial position of thistriangle and the desired material properties of the surface.

Thus, the parameter set includes the spatial coordinates (x_(s), y_(s),z_(s)) of a corner point of the triangle, the edge increments Δlu, Δru,Δld, Δrd, the number of scanlines n₁, n₂ of the triangle, and theinclination Δzx of the triangle surface with respect to the X axis andthe inclination Δy of the triangle surface with respect to the Y axis.The parameter set further includes information regarding the color ofthe triangle surface, the absorption behavior, the transparency of thetriangle surface and the like. The parameter set also includes a U,Vcoordinate pair for each corner point of the triangle, which pairdefines the position of the respective corner point in the reliefstructure. The parameter set thus allocates each corner point of thetriangle a point of the relief structure.

The parameter set data that define the spatial position of the triangleare supplied to a first assembly 1, which first divides the trianglesurface into a plurality of parallel scanlines, then divides eachscanline into a plurality of pixels. The spatial coordinates aresubsequently calculated for each pixel. The description of FIGS. 3a and3b includes details about the division of a triangle into pixels and thecalculation of the spatial coordinates of the individual pixels.

The parameter set present at the circuit input is also supplied to asecond assembly 2, which uses it to calculate the surface normal (n_(x),n_(y), n_(z))^(T) of the triangle surface. The second calculation unit 2also uses the predetermined position of the viewer and the position(x_(s), y_(s), z_(s)) of the triangle to calculate the viewing vector(e_(x), e_(y), e_(z))^(T), that is, the viewing direction of the viewer.

The parameter set is further supplied to a third assembly 3, which usesthe U, V coordinates of the corner points of the triangle to interpolatethe U, V coordinates of the individual pixels. Therefore, the associatedpoint in the relief structure is calculated for each pixel.

A matrix M is also calculated, which maps the spatial coordinates X, Y,Z of the individual pixels onto the U, V coordinates of the respectivepixels: ##EQU11##

As already described above, the parameter set includes the U, Vcoordinates for each corner point of the triangle. Because the X, Y, Zcoordinates of the three corner points are also known, an equationsystem of six equations is formed; the system has a total of sixunknowns M_(ij), and can be solved simply with the use of Cramer's Rule.

The matrix M serves, on the one hand, in rotating the relief structureparallel to the polygon surface, and--as will be described in detailbelow--on the other hand, in calculating a further matrix that orientsthe inclination vector with respect to the surface normal of the polygonsurface, the vector resulting from the relief structure.

The U, V coordinates obtained in this manner for each pixel are suppliedto an allocation unit 5, which allocates to each value pair (U, V) arelief value that defines the relief structure of the surface, and atexture value that defines the texture of the surface.

The allocation element has two two-dimensional storage matrices, namelythe texture matrix and the relief matrix, which are respectivelyaddressed by a U, V value pair.

The texture matrix defines a planar pattern that is superposed onto thesurface of the triangle. Each U, V value pair addresses an element ofthe texture matrix that determines the brightness value or color valueof the associated pixel on the surface of the triangle. The brightnessof the triangle surface is therefore not totally consistent, but canfluctuate from pixel to pixel, depending on the pattern stored in thetexture matrix. Consequently, the surface does not appear totallyconsistent, and therefore appears significantly more realistic. Thus,for example, the grain of a wood surface can be simulated. An importantpoint in this connection is that the texture matrix only includes thebrightness distribution or color distribution of the desired pattern,but does not simulate a relief structure.

The surface of the triangle therefore appears to be completely smooth.

The relief matrix, which, like the texture matrix, is addressed by theU, V value pair, serves in simulating a relief structure, that is, anarbitrary surface roughness.

The calculation of the relief matrix from a predetermined surfacestructure is explained in detail in the description of FIG. 4, using theexample of a brick wall.

A decisive factor in the simulation of the relief structure is theorientation of the local surface normal in the different pixels, becausethis determines the reflection behavior of the surface. The reliefmatrix therefore defines how the surface normal is inclined at theindividual points within the relief pattern with respect to the surfacenormal of the entire pattern. Hence, as can be seen from FIG. 4, thesurface normal does not extend perpendicular to the wall in the jointsof the brick wall, but at a diagonal. The relief matrix thereforedefines a value Δu, which determines the inclination of the localsurface normal with respect to the U axis, and a value Δv, whichdetermines the inclination with respect to the V axis.

Because each pixel is mapped onto a U, V value pair, the relief matrixdefines the relief structure to be impressed onto the surface of thetriangle.

The circuit further includes a third assembly 4, which uses the surfacenormal (n_(x), n_(y), n_(z))^(T) and the local inclination values Δu andΔv to calculate the pixel normal, that is, the local surface normal(n_(x) ', n_(y) ', n_(z) '), at the location of each pixel.

To this end, this calculation unit multiplies the differences Δu and Δvby a mapping matrix A, and adds the product to the surface normal(n_(x), n_(y), n_(z))^(T) : ##EQU12##

The circuit of the fourth calculation unit is shown in detail in FIG. 2.

Corresponding to the relief structure stored in the relief matrix, thepixels of a triangle each have an inclined local surfacenormal--referred to hereinafter as pixel normal--so the brightness ofthe pixels can fluctuate within a triangle due to the angular dependencyof the reflection behavior. In contrast to the level texture image thatis likewise mapped onto the surface of the triangle, the brightness of apixel is not only dependent on the position of the pixel within thetriangle surface, but is also dependent on the position of the viewerand the light sources relative to the triangle surface. If the viewerchanges positions, the perception of the relief structure changescorrespondingly, while the perception of the texture image remainsunchanged.

The local brightness value is calculated by a fourth assembly 6, whichreads the respective value of the reflection factor out of athree-dimensional reflection matrix as a function of the position of theviewer, as determined by the viewing vector (e_(x), e_(y), e_(z))^(T)and the pixel normals (n_(x) ', n_(y) ', n_(z) ').

In this connection, a distinction must be made between diffusereflection at the pixel and mirror reflection at the pixel. Athree-dimensional reflection matrix 7 is provided for calculating thereflection factor for diffuse reflection, and a furtherthree-dimensional reflection matrix 8 is provided for calculating thereflection factor for mirror reflection.

In diffuse reflection, it is assumed that the light is radiateduniformly in all direction, whereas in mirror reflection the radiationis effected according to the laws of geometrical optics, with the angleof incidence and the reflection angle being identical. Conversely, if a"line of sight" originating from the viewer is observed, rather than alight beam being emitted by the light source, this "line of sight" isreflected at the pixel in the direction of a reflection vector R. Thepixel normal, the reflection vector and the viewing direction thereforelie in a plane, with the angle between the pixel normal and the viewingdirection being identical to the angle between the reflection vector andthe pixel normal.

The following applies for the reflection vector as a function of thepixel normal N_(U) and the viewing direction, that is, the "line ofsight" E: ##EQU13## or, written in terms of components: ##EQU14##

The reflection matrix can be regarded as a cubic envelope surface, inthe center of which the pixel is located. FIGS. 5a and 5b show this typeof cubic envelope surface. Here a value of the reflection factor isassociated with each surface element of the envelope surface.

Therefore, in order to calculate the diffuse reflection factor, it isnecessary to determine the surface element of the cubic envelope surfacethat is intersected by the pixel normal or its extension. The valueassociated with this surface element is then the diffuse reflectionfactor.

Likewise, to calculate the mirror reflection, it is necessary todetermine the point of the envelope surface at which the reflectionvector or its extension penetrates the envelope surface. The valueassociated with this point of the envelope surface then defines thereflection factor in mirror reflection.

The values on the envelope surface thus define the illuminationenvironment of the triangle, and correspond to an all-around recordingof the environment from the perspective of the triangle. It is thereforeadvantageously possible to simulate a plurality of light sources. It isalso possible to simulate arbitrarily-distributed light sources.Furthermore, the viewer can change his position randomly, that is,travel through virtual scenery, in which instance the viewing impressionof the viewer is automatically adapted to the changed perspective.

As already described above, the cubic envelope surface is respectivelysimulated by a three-dimensional reflection matrix 7 or 8. The firstdimension of the reflection matrix can assume values between 1 and 6,and indicates which side surface of the envelope surface is penetratedby the pixel normals or the reflection vector.

The other two dimensions of the reflection matrix are addressed as afunction of the position of the point of intersection on the respectiveside surface. If the pixel normal or the reflection factor penetrates,for example, a side surface of the cubic envelope surface that isparallel to the X-Z plane, the Y coordinate of the point of intersectionaddresses the second dimension, and the Z coordinate of the point ofintersection addresses the third dimension of the reflection matrix. Theaddressing is not effected directly by the spatial coordinates of thepoint of intersection, but indirectly through the interposing of afurther calculation unit 9 or 10, which calculates the angle withrespect to the axis of the coordinate system penetrating the sidesurface in two perpendicular planes.

The calculation of the reflection factors for diffuse reflection andmirror reflection is described in detail in the description of FIGS. 5aand 5b, respectively.

A reflection factor r_(d) or r_(s) for diffuse reflection and mirrorreflection, respectively, is read out of the reflection matrix 7 or 8.Moreover, the allocation unit 5 provides a texture value t for eachpixel. The two reflection factors r_(d) and r_(s) and the texture valuet are supplied to a fade-in unit 11, which uses them and the color valuec of the triangle surface to calculate a brightness value h.

At its output, the circuit then emits the spatial coordinates x_(i),y_(i), z_(i) of the pixel, as well as the calculated brightness value h.

For display on a screen, the spatial coordinates of the pixel need onlybe converted into screen coordinates.

FIG. 2 shows the internal construction of the fourth calculation unit.The fourth calculation unit essentially comprises six multipliers 12.1through 12.6, and three adders 13.1 through 13.3, which use theinclination components Δu and Δv, the transformation matrix A and thecomponents n_(x), n_(y), n_(z) to calculate the local surface normal(pixel normal) for each pixel of the triangle: ##EQU15##

For this purpose, the coefficients A_(ij) of the transformation matrixand respectively one of the inclination components Δu or Δv are presentat the inputs of the multipliers 12.1 through 12.6. The output signalsdxv, dxu, dyv, dyu, dzv, dzu of the multipliers 12.1 through 12.6 aresupplied, along with the components n_(x), n_(y), n_(z) of the surfacenormal of the triangle, to the adders 13.1 through 13.3, which use themto calculate the components n'_(x), n'_(y), n'_(z) of the pixel normal.

As already mentioned in the description of FIG. 1, the input of thecircuit receives a parameter set that includes, among other information,the spatial position of the triangle to be represented. FIGS. 3a and 3billustrate the significance of these parameters by way of example.

The parameter set also includes the spatial coordinates x_(s), y_(s),z_(s) of a corner point P_(start) of the triangle. This corner pointserves as the starting point, and defines the position of the trianglein space, while the edge increments Δlu, Δru, Δld, Δrd define the shapeof the triangle and the position of the triangle relative to thecoordinate axes.

Starting from the starting point P_(start), the triangle is divided intoscanlines, and each scanline is in turn divided into pixels. Theindividual scanlines respectively extend parallel to the X axis. Theedge increment Δlu or Δld defines the inclination, relative to the Yaxis, of the triangle edge to the left of the starting point, while theedge increment Δru or Δrd defines the inclination, relative to the Yaxis, of the triangle edge on the right. In this way, the spatialcoordinates of the initial and end points of each scanline can becalculated.

Because the scanlines extend parallel to the X axis, the Y coordinate ofboth the initial point and the end point of each scanline results fromthe Y coordinate y_(s) of the starting point and the number of thescanline:

    y.sub.1 =y.sub.s +i y.sub.r =y.sub.s +i

The X coordinate x₁ of the initial points of each scanline resultsaccordingly from the X coordinate x_(s) of the starting point P_(start),the number i of the scanline and the edge increment Δlu or Δld. The edgeincrement Δlu is used within the first n₁ scanlines, and the edgeincrement Δld is used subsequently.

    x.sub.1 =x.sub.s +Δlu·i for i≦n,

    x.sub.1 =x.sub.s +Δlu·n.sub.1 +(i-n.sub.1)·Δld for n.sub.1 <i≦n.sub.1 +n.sub.2

In the same way, the X coordinate x_(r) of the end point of eachscanline results from the X coordinate x_(s) of the starting pointP_(start), the number i of the scanline and the edge increment Δru orΔrd. In this case, the edge increment Δru is used during the first n₁scanlines, and the edge increment Δrd is used subsequently:

    x.sub.r =x.sub.s +Δru·i for i≦n.sub.1

    x.sub.r =x.sub.s +Δru·n.sub.1 +(i-n.sub.1)·Δrd for n.sub.1 <i≦n.sub.1 +n.sub.2

The Z coordinate z₁ of the initial point results from the Z coordinatez_(s) of the starting point, the number of the scanline and theinclinations of the triangle surface with respect to the X and Y axes:

    z.sub.1 =z.sub.s +i·Δzy+(x.sub.1 -x.sub.s)·Δzx

The scanlines are then divided into pixels, with the spatial coordinatesfor each pixel being calculated from the known coordinates of theinitial and end points of the scanlines.

Because the scanlines extend parallel to the X axis, the Y coordinate ofeach pixel is identical to the Y coordinate of the associated scanline:

    y.sub.j =y.sub.1

The X coordinate of each pixel results from the X coordinate x₁ of theinitial point of the scanline, and the number j of the pixel in thescanline:

    x.sub.j =x.sub.1 +j

The Z coordinate of a pixel results from the Z coordinate z₁ of theinitial point of the associated scanline, the number j of the pixelwithin the scanline and the inclination Δzx of the triangle surfacerelative to the X axis.

    z.sub.j =z.sub.1 +j·Δzy

In this way, the spatial coordinates x_(j), y_(j), z_(j) are calculatedfor each pixel of the triangle surface.

FIG. 4 shows the surface of a wall comprising bricks 15, and theinclination values Δu and Δv that respectively result from the reliefstructure of the surface.

The joints 14 between the bricks 15 represent a groove-shaped depressionin the surface of the brick wall. The brick wall is thus not exactlylevel, but possesses a relief structure.

A cross-section of the surface along the line l_(u) is shown beneath thediagram. The upper line shows the course of the height H of the surfacealong the line l_(u). The groove-shaped depressions between the bricks15 can be seen clearly. Below this line is the inclination component Δuresulting from this surface course. The inclination component Δutherefore results from partial differentiation of the height level H:##EQU16##

Finally, the local surface normal is shown below this line.

On the right sides, the surface profile of the brick wall is shown ascross-section along the line l_(v). Shown next to this is theinclination component Δv, which results from this height profile and iscalculated through partial differentiation of the height level:##EQU17##

The inclination components Δu, Δv determine the inclination of the localsurface normal with respect to the surface normal of the ideally-smoothsurface.

Because a partial differentiation requires a relatively large amount oftime, the corresponding values for the desired relief structure--in thiscase the brick wall--are calculated in advance and stored in atwo-dimensional matrix, the so-called relief matrix. Thus, a value ofthe two inclination components Δu and Δv is allocated to each point ofthe brick wall.

Determining the inclination components Δu, Δv only requires one readaccess to the relief matrix that is addressed by the U, V value pair.

The cube illustrated in FIG. 5a serves in calculating the reflectionfactor in diffuse reflection at the pixel.

In diffuse reflection, the light is radiated uniformly in alldirections, as explained above. The brightness of a pixel is thereforenot dependent on the perspective of the viewer, but is determinedexclusively by the position of the pixel normal relative to theillumination environment.

For calculating the reflection factor, the cube is positioned such thatthe relevant pixel lies in the center point of the cube. The cube thusforms an envelope surface around the pixel, with a value for thereflection factor being allocated to each surface element.

For calculating the reflection factor, the point of intersection 17 ofthe pixel normal N_(U) and the envelope surface is determined, and theassociated value of the envelope surface is defined as the reflectionfactor.

The values on the envelope surface therefore define the illuminationenvironment of the triangle 16, and correspond to an all-around.recording of the environment from the perspective of the triangle 16. Itis therefore advantageously possible to simulate a plurality of lightsources. The simulation of arbitrarily-distributed light sources is alsopossible.

The cube is embodied here as a three-dimensional matrix, with the firstdimension of the matrix indicating which side surface of the cube isintersected by the pixel normal N_(U), while the two other dimensionsrepresent the position of the point of intersection 17 within the sidesurface.

Which envelope surface is penetrated by the pixel normal N_(U) --inother words, the addressing of the first dimension of the matrix--issolely dependent on the relationship of the individual components of thepixel normal N_(U). For example, if the X component is larger in amountthan the two other components and greater than zero, the pixel normalN_(U) penetrates the side surface of the envelope surface that isintersected at a right angle by the positive branch of the X axis, andthe first dimension of the reflection matrix correspondingly assumes thevalue of 1.

In general, the following results for the addressing value a₁ of thefirst dimension of the reflection matrix as a function of the componentsn_(x), n_(y), n_(z) of the pixel normal N_(U) : ##EQU18##

The other two dimensions a₂, a₃ of the reflection matrix are addressedas a function of the position of the point of intersection 17 on therespective side surface. If the pixel normal or reflection factorpenetrates, for example, a side surface of the cubic envelope surfacethat extends parallel to the X-Z plane, the Y coordinate of the point ofintersection addresses the second dimension, and the Z coordinate of thepoint of intersection addresses the third dimension of the reflectionmatrix.

In general, the following applies for addressing the other twodimensions a₂, a₃ of the reflection matrix: ##EQU19##

FIG. 5b shows a further cube, which serves in calculating the reflectionfactor in mirror reflection.

In mirror reflection at a surface, the reflection is not uniform in alldirections, as was assumed in the above-described diffuse reflection,but in a preferred direction of light reflection. According to the lawsof geometric optics, this direction of light reflection results from theposition of the surface normal and the direction of light incidence. Alight beam originating from a light source thus impacts the surface, andis reflected at the surface in the direction of light reflection, withthe angles of incidence and reflection being identical with respect tothe surface normal. Conversely, if the "line of sight" originating fromthe viewer is observed instead of the light beam originating from thelight source, this "line of sight" is reflected at the surface accordingto the same laws, and finally impacts an object of the environment. Inthis way, it can be determined which part of the environment isperceived by a viewer when mirror reflection occurs at a pixel.

For this purpose, a reflection vector Ru, which corresponds to the "lineof sight" reflected at the pixel. Because the angles of incidence andreflection are identical in mirror reflection, the following results forthe reflection vector R_(U) from the pixel normal N_(U) and the viewingvector E: ##EQU20## or, written in terms of components: ##EQU21##

As in the calculation of the reflection factor in diffuse reflection,the cube is disposed such that the relevant triangle 18 lies in thecenter of the cube. The cube therefore forms an envelope surface thatsurrounds the pixel, with a value of the reflection factor beingallocated to each surface element of the envelope surface. Fordetermining the reflection factor, therefore, it is necessary todetermine the surface element of the envelope surface that is penetratedby the reflection vector R_(U) or its extension. The value associatedwith this surface element is then the reflection factor.

The cubic envelope surface thus represents an all-around recording ofthe environment from the perspective of the pixel.

The cubic envelope surface is simulated by a three-dimensional matrix,as in diffuse reflection. Addressing of the first dimension is effecteddepending on which side surface is penetrated by the reflection factor.The following applies for the addressing value a₁ of the first dimensionof the matrix: ##EQU22##

The addressing of the other two dimensions of the matrix is effected asa function of the position of the point of intersection 19 within theside surface that is penetrated by the reflection vector R_(U). Thefollowing applies for the addressing values a₂, a₃ : ##EQU23##

Depending on the addressing values a₁, a₂, a₃ l the reflection factorfor mirror reflection is then read out of the matrix, and is consideredin the calculation of the pixel brightness, along with the reflectionfactor for diffuse reflection and the texture value of the pixel.

An advantage here is that the perspective of the viewer can be changedwithout necessitating the re-calculation of the reflection matrix. Theviewer can therefore change his perspective relative to the model in areal-time representation.

FIG. 6 shows a block diagram of a circuit for addressing athree-dimensional matrix with the use of a vector, with thethree-dimensional matrix simulating a cubic envelope surface thatsurrounds the origin of the vector.

The illustrated circuit is therefore suited for calculating thereflection factors for diffuse or mirror reflection with the cubes shownin FIGS. 5a and 5b, respectively.

Moreover, the circuit can be used wherever a vector addresses athree-dimensional matrix that simulates a cubic envelope surface, forexample, in place of the addressing unit 6, the calculation unit 9 or 10and the matrix 7 or 8 in FIG. 1.

At its input, the circuit receives the spatial coordinates x_(N), y_(N),z_(N) of the addressing vector.

The circuit first determines which side surface of the cubic envelopesurface is intersected by the vector. For this purpose, the largestspatial coordinate of the addressing vector is determined. If, forexample, the X coordinate of the vector is the largest, the vectorintersects the side surface of the cubic envelope surface that extendsparallel to the Y-Z plane.

The spatial coordinates x_(N), y_(N), z_(N) are therefore supplied to arespective calculation unit 20.1, 20.2, 20.3, which forms the absolutevalue of the spatial coordinates. The absolute values of the spatialcoordinates--in other words, the output signals of the calculation units20.1, 20.2, 20.3--are subsequently supplied in pairs to threecomparators 21.1, 21.2, 21.3, which compare the absolute values of thespatial coordinates.

The output signals of the comparators 21.1, 21.2, 21.3 are then suppliedto an addressing unit 22, which calculates the addressing value a₁ forthe first dimension of the three-dimensional matrix 27. In the process,the addressing unit 22 compares the output signals of the comparators21.1, 21.2, 21.3, and uses the result and the operational signs tocalculate the addressing value a₁ according to the following rule:##EQU24##

The circuit further determines the position at which the addressingvector penetrates the side surface of the cubic envelope surface. Thecoordinates of this point of intersection serve in addressing the secondand third dimension of the matrix 27.

To this end, the circuit forms the quotient from one of the spatialcoordinates that is smaller with respect to amount, and the largestspatial coordinate. The addressing unit therefore generates a selectionsignal Sel, which is supplied to the multiplexers 23.1, 23.2, 23.3.

The multiplexer 23.3 selects the largest spatial coordinate, andsupplies its value to the calculation unit 26, which uses it to form thereciprocal value, and outputs it separately as mantissa and exponent.

The two other multiplexers 23.1, 23.2 respectively select one of the twosmaller spatial coordinates as counters.

The output signals of the multiplexers 23.1, 23.2 are then respectivelysupplied, along with the mantissa of the reciprocal value of the largestspatial coordinate, the mantissa having been determined by thecalculation unit 26, to a multiplier 24.1, 24.2, which multiplies thesmaller spatial coordinates by the mantissa of the reciprocal value ofthe largest spatial coordinate.

The output signal of the multipliers 24.1, 24.2 is supplied to two shiftregisters 25.1, 25.3, which perform multiplication with the exponentscalculated by the calculation unit 26.

The following applies for the output signals a₂, a₃ of this addressinglogic, for example if x_(N) is the component having the largest value:##EQU25##

The output signals a₂, a₃ then address the second or third dimension,respectively, of the matrix 27, which is embodied, for example, as asemiconductor memory.

The circuit illustrated in FIG. 7 serves in calculating the inclinationof the pixel normal that is caused by the simulated relief structure, asa function of the inclination components Δu, Δv in the U, V coordinatesystem.

The circuit receives, as input values, the four components i, j, k, s ofthe quaternion, as well as the inclination components Δu, Δv in the U, Vcoordinate system. The task of the quaternion is to orient theinclination vector determined in the U, V coordinate system with respectto the pixel normal of the smooth surface in the X, Y, Z coordinatesystem. The inclination vector must be oriented at a right angle to thepixel normal in the X, Y, Z coordinate system, so that the addition ofthese two vectors results in an inclination of the pixel normals.

At its output, the circuit then outputs the components dx', dy', dz' ofthe inclination vector that must be added to the pixel normals.

The components dx', dy', dz' of the inclination vector are calculatedaccording to the following formulas:

    dx'=Δu·(s.sup.2 -k.sup.2 +i.sup.2 -j.sup.2)+2·Δv·(i·j+k·s)

    dy'=Δv·(s.sup.2 -k.sup.2 +i.sup.2 +j.sup.2)+2·Δu·(i·j-k·s)

    dz'=2·Δu·(i·k+j·s)+2·.DELTA.v·(j·k-i·s)

For performing this task, the circuit has 16 multipliers, 6 adders and 5subtracters, which perform the calculation operations shown above asformulas.

The diagram shown in FIG. 8 clarifies the scaling of a relief structurewith the use of a scaling factor BS (Bump Scale). The scaling factor BS,which permits a continuous setting of the relief effect of the reliefstructure simulated on the object's surface, is shown on the abscissa.Correspondingly, a value D, which represents the spatial depth of therelief structure appearing on the object's surface, is shown on theordinate.

To explain the dependency, the three-dimensional relief-structureformers are shown for four values of the scaling factor BS; theseformers result from the respective value of the scaling factor BS. Itcan be seen clearly that the relief effect of the relief-structure imageappearing on the object's surface increases with the scaling factor BSin the representation. This is achieved in that the inclination valuesthat have been read out of the relief matrix are first multiplied by thescaling factor BS before the course of the surface is calculated fromthe local surface normal of the smooth surface and the respectiveinclination values, taking into consideration the relief structure.

The invention is not limited in its embodiment to the above-described,preferred embodiments. Rather, numerous variations are conceivable thatmake use of the illustrated solution, even in fundamentally differentembodiments.

What is claimed is:
 1. An image-processing method for simulating arelief structure in representing objects that particularly comprise atleast one polygon surface, in whicha first assembly of a calculationunit uses a first parameter set representing the position of the polygonsurface to calculate a plurality of coordinate sets (x, y, z, u, v),which represent the position of a respective pixel of the polygonsurface, and, for each coordinate set (x, y, z, u, v),a second assemblyuses the first parameter set to calculate a second parameter set (n_(x),n_(y), n_(z)), which represents the position of the surface normal ofthe polygon surface, without taking into consideration the reliefstructure, a third assembly uses the second parameter set (n_(x), n_(y),n_(z)) and an inclination value (du, dv) to calculate a third parameterset (n'_(x), n'_(y), n'_(z)), which represents the position of the localsurface normal of the polygon surface, taking into consideration therelief structure, a fourth assembly determines a value (h) thatrepresents the brightness or the color of a pixel, and creates theimpression of a relief structure, as a function of the third parameterset (n'_(x), n'_(y), n'_(z)), characterized in that a storage locationof a first storage matrix is addressed by each coordinate set (x, y, z,u, v), each storage location of the first storage matrix contains aparameter set that represents the inclination of the local surfacenormal, taking into consideration the relief structure, relative to thesurface normal of the polygon surface, without taking into considerationthe relief structure, and the inclination value (du, dv) is read out ofthe first storage matrix.
 2. The method according to claim 1,characterized in that the polygon surface is planar, and the secondparameter set (n_(x), n_(y), n_(z)) represents the spatial position ofthe surface normal of the entire polygon surface.
 3. The methodaccording to claim 1, characterized in that the polygon surface iscurved, and the first parameter set represents the spatial position ofthe polygon surface and the corner-point normals,and the secondparameter set (n_(x), n_(y), n_(z)) is calculated from the firstparameter set and a respective coordinate set (x, y, z, u, v), andrepresents the spatial position of the local surface normal in arespective pixel.
 4. The method according to claim 1, characterized inthat the fourth assembly uses the third parameter set (n'_(x), n'_(y),n'_(z)), and a predetermined fourth parameter set (e_(x), e_(y), e_(z))that represents the perspective, and/or a predetermined fifth parameterset that represents the illumination conditions, to calculate the value(h) that represents the brightness and/or the color of a pixel andcreates the impression of a relief structure.
 5. The method according toclaim 1, characterized in thatthe fourth assembly uses the thirdparameter set (n'_(x), n'_(y), n'_(z)) and the fourth parameter set(e_(x), e_(y), e_(z)) to calculate two angular values for eachcoordinate set (x, y, z, u, v), which values represent the position ofthe local surface normal of the polygon surface, taking intoconsideration the relief structure, relative to the viewing direction ofthe viewer, a second storage matrix is addressed by the two angularvalues, each storage location of the second storage matrix contains avalue that represents the brightness of the polygon surface for aspecific perspective, and the value (h) that represents the brightnessand/or the color is read out of the second storage matrix.
 6. The methodaccording to claim 1, characterized in that a third storage matrix thatrepresents a cubic envelope surface surrounding a pixel is addressed bythe third parameter set (n'_(x), n'_(y), n'_(z))a first reflection value(r_(d)) representing the brightness of a pixel caused by diffusereflection is read out of the third storage matrix, and the value (h)representing the brightness or the color is calculated as a function ofthe first reflection value (r_(d)).
 7. The method according to claim 1,characterized in thatan addressing value representing the spatialposition of a light beam (R_(U)), which reaches the viewer followingmirror reflection at the pixel, is calculated from the third parameterset (n'_(x), n'_(y), n'_(z)), the fourth parameter set (e_(x), e_(y),e_(z)) and a respective coordinate set (x, y, z, u, v), a fourth storagematrix), which represents a cubic envelope surface that surrounds apixel, is addressed by the addressing value, a second reflection value(r_(s)) that represents the brightness of a pixel caused by mirrorreflection is read out of the fourth storage matrix, and the value (h)representing the brightness and/or the color is calculated as a functionof the second reflection value (r_(s)).
 8. The method according to claim1, characterized in that the polygon surface is triangular.
 9. Themethod according to claim 1, characterized in that a polygon should beunderstood to be an object whose surface structure is reduced, that is,less detailed, than the fine surface structure to be represented. 10.The method according to claim 1, characterized in that, for setting therelief effect of the relief structure to be simulated, a multipliermultiplies the inclination value (du, dv) by a predetermined scalingfactor (BS) after the inclination value has been read out of the firststorage matrix.
 11. An image-processing apparatus for executing themethod according to claim 1, having a calculation unit that includesaninput for receiving a first parameter set that represents the positionof the polygon surface, a first assembly that uses the first parameterset to calculate a plurality of coordinate sets (x, y, z, u, v) thatrepresent the position of a respective pixel, a second assembly forcalculating a second parameter set (n_(x), n_(y), n_(z)) for eachcoordinate set (x, y, z, u, v), the parameter set representing theposition of the surface normal of the polygon surface, without takinginto consideration the relief structure, a third assembly that uses thesecond parameter set (n_(x), n_(y), n_(z)) and an inclination value (du,dv) to calculate a third parameter set (n'_(x), n'_(y), n'_(z)) for eachcoordinate set (x, y, z, u, v), the parameter set representing theposition of the local surface normal of the polygon surface, taking intoconsideration the relief structure, a fourth assembly for calculating avalue (h) for each coordinate set (x, y, z, u, v), the valuerepresenting the brightness and/or the color of a respective pixel, as afunction of the third parameter set (n'_(x), n'_(y), n'_(z)),characterized in that a first storage matrix having a plurality ofstorage locations is provided for determining the inclination value (du,dv), each storage location of the first storage matrix contains aparameter set (du, dv) that represents the inclination of the localsurface normal of the polygon, surface, taking into consideration therelief structure, with respect to the surface normal of the polygonsurface, without taking into consideration the relief structure, and afirst addressing unit is provided for addressing the first storagematrix as a function of a respective coordinate set (x, y, z, u, v). 12.The apparatus according to claim 11, characterized in thatthe fourthassembly has a second addressing unit for calculating two angular valuesfrom the third parameter set (n'_(x), n'_(y), n'_(z)) and the fourthparameter set (e_(x), e_(y), e_(z)), with the two angular valuesrepresenting the position of the local surface normal, taking intoconsideration the relief structure, relative to the viewing direction ofthe viewer, the fourth assembly has a second storage matrix, which isaddressed by the two angular values, and each storage location of thesecond storage matrix contains a predetermined value that represents thebrightness or the color of the polygon surface for a certainperspective.
 13. The apparatus according to claim 11, characterized byathird storage matrix having a plurality of storage locations, which eachstorage location containing a reflection value (r_(d)) that representsthe brightness of a respective pixel that is caused by diffusereflection, and a third addressing unit for addressing the third storagematrix as a function of the third parameter set (n'_(x), n'_(y),n'_(z)), and for reading out the reflection value (r_(d)).
 14. Theapparatus according to claim 11, characterized bya fourth addressingunit that uses the third parameter set (n'_(x), n'_(y), n'_(z)), thefourth parameter set (e_(x), e_(y), e_(z)) and a respective coordinateset (x, y, z, u, v) to calculate an addressing value that represents thespatial position of a light beam that reaches the viewer followingmirror reflection at the pixel, and a fourth storage matrix that isdisposed downstream of the fourth addressing unit and can be addressedby the addressing value, the matrix containing values (r_(s)) thatrepresent the brightness of the pixels that is caused by mirrorreflection.
 15. The apparatus according to claim 13, characterized inthat the fourth assembly has a fade-in unit for calculating the value(h) that represents the brightness and/or the color, as a function ofthe first reflection value (r_(d)) and/or the second reflection value(r_(s)).
 16. The apparatus according to claim 11, characterized in thata multiplier is disposed downstream of the first storage matrix and thethird assembly for scaling the inclination values (du, dv) in order toset the relief effect.
 17. The apparatus according to claim 11,characterized in that a plurality of storage matrices is provided forsimulating a dynamic relief structure, the matrices respectivelycontaining the inclination values (du, dv) of an instantaneous recordingof the relief structure, and a selection unit is provided for sequentialselection of a respective storage matrix serving in simulating a reliefstructure.
 18. The apparatus according to claim 1, characterized in thata plurality of storage matrices is provided for dynamic texturesimulation, the matrices containing color or brightness valuesrepresenting an instantaneous recording of the texture image,and aselection unit is provided for sequential selection of a respectivestorage matrix serving in simulating a texture.
 19. The apparatusaccording to claim 1, characterized in that the storage matrix servingin texture simulation is connected to a video- or computer-graphicssystem for generating a dynamic texture image.